Add the content-stream tokeniser and inline images - #4
Merged
Conversation
A page's marking operations, and the one thing in a content stream that cannot be read by looking at it. Operators come back with their operands, and rubbish in the middle costs the byte it occupies rather than everything after it, which is what a renderer needs from a corpus where a handful of producers write bytes no grammar allows. Where an inline image's data stops is genuinely ambiguous: the bytes between ID and EI are raw, and compressed bytes do spell EI by accident. Three answers are tried in order of how far they can be trusted. An image with no filter states its own length exactly through width, height, depth and colour space. A declared /L is believed when an EI really does follow it. Failing both, every EI in the data is tried and the first whose data actually decodes — through the declared filter, to the number of samples the image claims, and for a JPEG with the markers a JPEG begins and ends with — is the one taken. Scanning alone gets this wrong, and the corpus proves it: one page of 1023 inline images turned into binary rubbish until the candidates were made to decode before being believed. The number lexer now stops at the first byte that cannot continue a number rather than swallowing the whole run of regular characters. Producers write "3.4-5" for two numbers, and twenty-one pages in the corpus were losing everything after that point. The same change makes "12x" two tokens instead of an error, which is what every other reader does. Measured over the whole corpus: 1 536 769 753 operations across 138 337 pages in ninety seconds, no panics, no page failing to decode, 15 854 inline images read, and seventy distinct operators — every one of them real, where before these two fixes there were ninety-nine and the extra twenty-nine were image data being read as code. Exact 100% statement coverage, go vet clean, nine cross-compilation targets. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A page's marking operations, and the one thing in a content stream that cannot
be read by looking at it. Operators come back with their operands, and rubbish
in the middle costs the byte it occupies rather than everything after it, which
is what a renderer needs from a corpus where a handful of producers write bytes
no grammar allows.
Where an inline image's data stops is genuinely ambiguous: the bytes between ID
and EI are raw, and compressed bytes do spell EI by accident. Three answers are
tried in order of how far they can be trusted. An image with no filter states
its own length exactly through width, height, depth and colour space. A
declared /L is believed when an EI really does follow it. Failing both, every
EI in the data is tried and the first whose data actually decodes — through the
declared filter, to the number of samples the image claims, and for a JPEG with
the markers a JPEG begins and ends with — is the one taken. Scanning alone gets
this wrong, and the corpus proves it: one page of 1023 inline images turned
into binary rubbish until the candidates were made to decode before being
believed.
The number lexer now stops at the first byte that cannot continue a number
rather than swallowing the whole run of regular characters. Producers write
"3.4-5" for two numbers, and twenty-one pages in the corpus were losing
everything after that point. The same change makes "12x" two tokens instead of
an error, which is what every other reader does.
Measured over the whole corpus: 1 536 769 753 operations across 138 337 pages
in ninety seconds, no panics, no page failing to decode, 15 854 inline images
read, and seventy distinct operators — every one of them real, where before
these two fixes there were ninety-nine and the extra twenty-nine were image
data being read as code.
Exact 100% statement coverage, go vet clean, nine cross-compilation targets.